- Introduction
- Research synthesis
- Comparison of meta-analysis and BES
- Meta-analysis
- Bayesian evidence synthesis
- Simulation
- Data generation
- Sampling population effect sizes
- Hypotheses
- Simulation results I
- Simulation results II
21 January 2022
META-ANALYSIS
Combines effect sizes
Asks what the average effect is in all studies combined (data-pooling method)
Effect sizes need to have the same metric and come from the same model
BAYESIAN EVIDENCE SYNTHESIS
Combines support for a specific hypothesis
Asks to what extent a hypothesis is supported in each study (NOT a data-pooling method)
There are no restrictions on which estimates can be combined
Computing meta-analytic effect size \(\bar{\beta}\) and its variance \(v\) (Hedges & Vevea 1998):
\[
\begin{aligned}
\bar{\beta} &= \frac{\sum_{k=1}^{K} w_k\hat{\beta_k}}{\sum_{k=1}^{K} w_k} \\
\\
w_k &= \frac{1}{var(\hat{\beta_k})} \\
\\
v &= \frac{1}{\sum_{k=1}^{K} w_k}
\end{aligned}
\]
where \(K\) is the number of studies, and \(\hat{\beta_k}\) is the estimated effect size of study \(k\).
\[
\pi_{K,h} = \frac{\prod_{k=1}^{K}\pi_{k,h}}{\sum_{h=1}^{H}\prod_{k=1}^{K}\pi_{k,h}}
\]
where \(h\) is hypothesis 1, …, \(H\) and \(\pi\) is the PMP.
Generate data from a regression model with one predictor: \[ Y = \beta X_1 + \epsilon, \] where \(X_1\) is sampled from \(\mathcal{N}(0,1)\) and \(\epsilon\) is sampled from from \(\mathcal{N}(0,\sqrt{1-R^2})\).
beta <- 0.3 # specify beta value R2 <- beta^2 # R-squared n <- 25 # sample size X <- rnorm(n = n, mean = 0, sd = 1) # sample X values Y <- X * beta + rnorm(n = n, mean = 0, sd = sqrt(1 - R2)) # compute Y
Instead of assuming one underlying population effect size, we assume study-specific population effect sizes that randomly deviate from the overall mean following a normal distribution \(\mathcal{N}(0.3,0.1)\):
We tested \(H1: \beta > 0\) against either the null hypothesis (\(H0\)), the complement hypothesis (\(Hc\)) or the unconstrained hypothesis (\(Hu\)):
\[
\begin{aligned}
H0&: \ \beta = 0 \\
Hc&: \ \beta < 0 \\
Hu&: \ \beta
\end{aligned}
\]
where \(Hu\) means that \(\beta\) can take on any value.
Proportion of simulations (nsim = 1000) that H1 gets selected over the alternative hypothesis based on the aggregated PMP:
Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376. https://doi.org/10.1038/nrn3502
Hedges, L. V., & Vevea, J. L. (1998). Fixed-and random-effects models in meta-analysis. Psychological Methods, 3(4), 486. https://doi.org/10.1037/1082-989X.3.4.486
Kuiper, R. M., Buskens, V., Raub, W., & Hoijtink, H. (2013). Combining statistical evidence from several studies: A method using Bayesian updating and an example from research on trust problems in social and economic exchange. Sociological Methods & Research, 42(1), 60-81. https://doi.org/10.1177/0049124112464867
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
Van Calster, B., Steyerberg, E. W., Collins, G. S., & Smits, T. (2018). Consequences of relying on statistical significance: Some illustrations. European Journal of Clinical Investigation, 48(5), e12912. https://doi.org/10.1111/eci.12912